don't panic when there are concurrent requests #1736
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Summary: Allow us to test the bandwidth of concurrent rdma operations. actual concurrency support will be added later in the stack. Differential Revision: D85724514
Summary: This diff adds a FIFO waiting mechanism for acquiring the queue pair. **True concurrency support will come in follow-up diff**: this will require we track wr_id and create a polling task for each queue pair, which will need further refactoring. **Limitations** - No true concurrency support: only one request can be in-flight at a time per queue pair connection. However, subsequent requests now wait fairly (FIFO) instead of panicking with "already checked out" errors. - request_queue_pair/release_queue_pair is still not cancel safe (if an operation times out, the queue pair might not be returned and subsequent requests will fail) Core changes: 1. **Fair Waiting via Semaphore**: Replaced Available/CheckedOut states with Connecting/Ready/ConnectionError. Added QueuePairEntry wrapping RdmaQueuePair + Arc<Semaphore>. request_queue_pair uses two-phase approach (get/create QP, then acquire semaphore permit for FIFO fairness). 2. **Refactor - Moved read_into/write_from to RdmaManagerActor**: Prevents deadlock in actor message queue. Old design had RdmaBuffer call request_queue_pair RPC → perform operation → call release_queue_pair RPC, causing release messages to queue behind waiting requests. Now entire operation (request → use → release) happens within single actor message handler. 3. **Refactored Connection Logic**: Extracted establish_connection helper handling both loopback and remote connections. 5. **Test**: Added create_buffer_pair method and concurrent tests. Differential Revision: D85627877
meta-cla
bot
added
the
CLA Signed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
This diff adds a FIFO waiting mechanism for acquiring the queue pair.
True concurrency support will come in follow-up diff: this will require we track wr_id and create a polling task for each queue pair, which will need further refactoring.
Limitations
Core changes:
Fair Waiting via Semaphore: Replaced Available/CheckedOut states with Connecting/Ready/ConnectionError. Added QueuePairEntry wrapping RdmaQueuePair + Arc. request_queue_pair uses two-phase approach (get/create QP, then acquire semaphore permit for FIFO fairness).
Refactor - Moved read_into/write_from to RdmaManagerActor: Prevents deadlock in actor message queue. Old design had RdmaBuffer call request_queue_pair RPC → perform operation → call release_queue_pair RPC, causing release messages to queue behind waiting requests. Now entire operation (request → use → release) happens within single actor message handler.
Refactored Connection Logic: Extracted establish_connection helper handling both loopback and remote connections.
Test: Added create_buffer_pair method and concurrent tests.
Differential Revision: D85627877